Process for computerized grading of formula-based multi-step problems via a web-interface

ABSTRACT

The method of computerized grading of formula-based multi-step problems includes the steps of providing a memory for storing problems and data to enable and facilitate assessment of correctness of problems, storing at least one formula-based multi-step problem in the memory, a complete solution to the problem requiring calculation of at least one intermediate answer and a subsequent calculation using said intermediate answer to arrive at said complete solution, providing a computer program that can grade submitted answers to the steps of said problem, providing a user interface with selective access to the memory and the computer program, and providing a global computer network with which the user interface can be connected to the memory and grading program.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to a process for computerizedgrading of formula-based multi-step problems via a web-interface. Atpresent, it is generally known to grade math, science, engineeringand/or technology problems via the use of a global computer networkthrough access to websites. Such services are currently being renderedby distance education providers such as Blackboard, Brownstone, WebCT,ecollege and ANGEL. Major textbook publishers have created websites forthe same purpose. Applicant is aware of publishers John Wiley and Sons,Brooks/Cole, Addison Wesley, and Prentice Hall establishing websites forthis purpose. Additionally, academic institutions have been offeringcomputerized grading of problems, both alone and in partnership withdistance education providers or computer algebra system (CAS) vendors.Applicant is aware that two major such vendors, Maple and Mathematica,have developed working relationships with distance education providers,academic institutions, and publishers.

[0002] Systems developed by Maple and Mathematica can manipulatemathematical formulas and can recognize any numerical or symbolicexpression that is likely to be an answer to any problem in any courseone may encounter at the pre-college, undergraduate, or graduate level.As a result, the most intense current activity in computer-grading ofmath-type problems involves grading answers that are symbolic ratherthan multiple choice type answers, fill-in-a-blank type answers orpurely numeric answers. The significant features employed by knownsymbolic answer graders are the following:

[0003] (1) A variety of methods for free-form entry ofsymbolic-expression answers;

[0004] (2) Gradable problems have parameters that can be used to changenumerical values in the problem statement. Assignment of parameters canbe random;

[0005] (3) The use of computer algebra systems to grade problems bycomparing a submitted answer to a pre-programmed correct answer, withchecks to ensure that an answer is of the correct type, for example,algebraic category, and that students submit an answer rather thancomputer-algebra-system instructions to achieve the answer;

[0006] (4) Intelligent hints to permit algorithmic, interactiveself-grading of an answer;

[0007] (5) Sub-questions employed as hints to the student if the studentwishes to use them;

[0008] (6) The ability to resubmit incorrect answers repeatedly until acorrect answer is achieved with provision for penalties for multipletries to correctly answer a problem;

[0009] (7) Use of predefined answer types such as, for example,algebraic, matrix, constant, multiple choice, or string;

[0010] (8) Systems are known that permit both students and instructorsto obtain detailed progress reports.

[0011] Distance education providers emphasize customization of problemsto meet the desires and needs of an individual instructor.

[0012] The problems that typically appear in course textbooks foralgebra, calculus, and other mathematical and technical subjects areformula-based multi-step problems for which answers to earlier steps areused to obtain subsequent intermediate and final answers. Moreover, aparticular intermediate answer may be obtainable from prior answers inseveral different ways, each of which may involve an equally validmethodology. Many, if not most, problems assigned for homework or givenon a quiz or exam in such courses are typically adapted from a textbookand hence are of this multi-step form.

[0013] As should be understood from the above explanation, currentweb-based systems grade only one answer at a time. Related answers arerequested diagnostically, for example, as a hint or sub-problem, onlyafter a single answer is recognized as incorrect. As such, in suchsystems, the student must submit all correct answers, one at a time, toobtain full credit.

[0014] By contrast, effective hand-grading of formula-based multi-stepproblems uses two shortcut-type strategies, referred to herein as thePrior-Answer Strategy and the Later-Answer Strategy:

[0015] (1) The Prior-Answer Strategy looks for correct intermediate orfinal answers, which when found give full credit for all relevant prioranswers used to get it.

[0016] If an incorrect intermediate answer is found, an experiencedhand-grader will use the second strategy;

[0017] (2) The second strategy is the Later-Answer Strategy in whichsubsequent steps are examined and subjective credit is given for answersthat are incorrect, but which appear to have been obtained correctlybased upon incorrect but reasonable answers to earlier steps.

[0018] When hand-graders apply these two strategies combined with theinnate human ability to recognize and accept more than one correctformula for using prior answers to obtain later answers, the studentobtains the maximum credit deserved for problem steps that werecorrectly performed. The prior art fails to teach or suggest anycomputerized system that incorporates these grading strategies used byeffective hand-graders to fairly and fully grade problems. It is withthis thought in mind, among others, that the present invention wasdeveloped.

SUMMARY OF THE INVENTION

[0019] The present invention relates to a process for computerizedgrading of formula-based multi-step problems via a web interface. Inaccordance with the teachings of the present invention, multi-answerproblems can be computer graded in a manner analogous to effectivehand-grading by a skilled teacher, but in a more consistent fashion andmore fairly and impartially than can a human being.

[0020] The present invention includes the following interrelatedobjects, aspects and features:

[0021] (1) The inventive system grades multi-answer problems that lookand feel like typical homework and test problems that instructorscurrently assign students and then grade by hand.

[0022] (2) Students can opt to submit some or all of a problem's answerssimultaneously. This is a distinct improvement over the prior art inwhich answers must be submitted sequentially.

[0023] (3) For each answer, the inventive system can recognize thecorrect use of several different formulas possibly involving differentprior answers and possibly even getting different numeric or symbolicvalues that can, under certain circumstances, be graded as correctanswers.

[0024] (4) In accordance with the teachings of the present invention,parameters used in problems and their solutions can be symbolic as wellas numeric, thereby greatly enhancing the versatility of a givenproblem.

[0025] (5) The present invention can employ the Prior-Answer Strategy orthe Later-Answer Strategy or both. In the Prior-Answer Strategy, onelooks for correct intermediate or final answers, which when found permitgiving of full credit for all unsubmitted prior answers used to get it.On the other hand, if a submitted answer is incorrect, the Later-AnswerStrategy examines subsequent steps and gives credit for answers that areincorrect but obtained correctly from incorrect but reasonable answersto earlier steps. In other words, if in an earlier step, the studentobtains a wrong answer, then that wrong answer will be carried throughto render subsequent answers wrong as well. However, if the incorrectanswer is reasonable (i.e., isn't inappropriate or overly simple) andthe student applies a proper process for answering subsequent partsbased upon the incorrect intermediate answer, the inventive systempermits giving at least partial credit for the subsequent answers thatare technically incorrect but properly follow a correct requiredprocess.

[0026] (6) The inventive system includes an answer entry window thatinforms the student of the minimal set of submitted answers required forfull credit. The answer entry window operates by displaying each step'sanswer entry area as “C” (already correct, hence disabled), “R”(required for full credit), or “O” (optional, but gradable). Of course,the letters C, R and O are merely exemplary and any desired letters orother visual identifiers can alternatively be employed. A non-C answeris displayed as O if it is used in one or more R-answer formulas. By thePrior-Answer Strategy, an unsubmitted O-answer will be graded as correctif an R-answer obtained using a formula in which it appears is graded ascorrect.

[0027] (7) The author of the problem must submit a single, easilyprepared computer-algebra-system procedure, herein designated asanswers( ) although any meaningful name can be used, containing only theacceptable formulas for each answer. This single procedure is used bothto prepare the C-R-O display as explained in paragraph

[0028] (6) above, and to implement the Prior-Answer and Later-AnswerStrategies.

[0029] (8) All data base information needed to grade problems is storedin the form of computer-algebra-system instructions. In this way,problems are easy to prepare for inclusion in assessment instrumentsconducted in accordance with the teachings of the present invention.

[0030] (9) In practicing the present invention, problems are uploaded toa problem table of the inventive system's data base via an authoringtool, to be described in greater detail hereinafter, which first teststhe author provided computer-algebra-system information for correctnessand accuracy. This ensures reliability of problems stored on the database.

[0031] (10) In this system, all problems are viewed as owned by a partythat may or may not be the problem author. The present invention isconfigurable so that in return for negotiated incentives, Copyrightowners will allow their problems to be freely available for use on asingle searchable data base, with each customer, whether university,sub-department or other institution, having selective access to adesired class of problems.

[0032] Accordingly, it is a first object of the present invention toprovide a process for computerized grading of formula-based multi-stepproblems via a web interface.

[0033] It is a further object of the present invention to provide such aprocess and system in which the Prior-Answer Strategy and Later-AnswerStrategy typically used by experienced hand-graders are programmedtherein.

[0034] It is a still further object of the present invention to providesuch a process and system which permits grading of multi-answer problemshaving the look and feel of typical homework and test problems thatinstructors currently assign and grade by hand.

[0035] It is a still further object of the present invention to providesuch a process and system that permits recognizing the correct use ofseveral different formulas, even possibly involving different prioranswers and getting different resulting answers.

[0036] It is a yet further object of the present invention to providesuch a process and system in which parameters can be symbolic as well asnumeric.

[0037] It is a still further object of the present invention to providesuch a process and system including incorporation of an answer entrywindow that tells the student the minimum set of submitted answers thatare required for full credit.

[0038] These and other objects, aspects and features of the presentinvention will be better understood from the following detaileddescription of the preferred embodiments when read in conjunction withthe appended drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0039]FIG. 1 shows a schematic representation of the inventive system.

[0040]FIG. 2 shows a graph depicting the problem of Example 1.

[0041]FIG. 3 shows a schematic representation of an answer entry windowfor a portion of the problem solving process for the problem of Example1.

[0042]FIG. 4 shows a schematic representation of an answer entry windowfor a further portion of the problem solving process for the problem ofExample 1.

[0043]FIG. 5 shows a graph depicting the problem of Example 2.

SPECIFIC DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] The inventive process grades multi-answer problems for whichcorrect answers can be symbolic and may be obtainable by severalformulas or methods. Its key innovative features are the following,which simulate effective hand grading of such problems. It can informthe student, based on what was previously graded as correct, of whichanswers are required, hence must be submitted for full credit;

[0045] It allows: the student to (re-)submit simultaneously, a selectedset of some or all required and/or optional answers that were not yetgraded as correct;

[0046] It can recognize several formulas or methods, which can involveboth symbolic and numeric parameters and may give different values, ascorrect processes for getting a particular answer;

[0047] It can use the Prior-Answer Strategy and Later-Answer Strategy,as explained in detail below, to give maximum deserved partial creditfor a given set of submitted answers.

[0048] These features can be implemented in a variety of ways. Theexamples described herein seek to emphasize understandability ratherthan efficiency or flexibility of implementation so as to explain thatwhich Applicant regards as the invention. To this purpose, the exampleproblems are math problems with parameters and answers that are eitherintegers, fractions, or algebraic expressions or equations. However, theinventive process can, just as effectively, be used to grade typicalproblems for any quantitative course, and for which problem parametersand answers can involve decimals and/or any symbolic objects that can berecognized by a CAS, e.g., trigonometric or boolean (true/false)expressions, vectors, matrices, graphs, differential equations, etc.

[0049] The author of a multi-answer problem must provide the text of theproblem statement, formulas that can be used to get correct answers, andrestrictions on the problem's parameters. Two examples are used below toillustrate this information.

[0050]FIG. 2 shows a graph of problem Example 1, an analytic geometryproblem with six answers (denoted in boldface as _A1, _A2, . . . , _A6in the problem text) and four integer parameters, xA, xB, yA, and yB.The delimiters |* *| in the problem text enclose a CAS expression thatis to be displayed in math notation (viz., as {square root}2) in theAnswer-Entry Window. Other delimiters can be used.

EXAMPLE 1 4-Parameter Analytic Geometry Problem

[0051] Problem Text

[0052] For the line segment from A(xA, yA) to B (xB, yB) in thexy-plane: It has x-increment Dx=_A1, slope m=_A2, length=_A3, andy-intercept b=_A4. They coordinate of its midpoint M(xM, yM) is yM=_A5,and an equation for the circle having AB as a diameter is _A6. Enterexact numbers, not decimals, e.g., ⅓, not 0.33; |* sqrt(2) *|, not1.414; etc.

[0053] Formulas for Answers [_Ai Denotes a Prior Answer Used to get _Aj]

[0054] For _A1: xB−xA. #=Dx; similarly Dy=yB−yA

[0055] For _A2: (yB−yA)/(xB−xA) or (yB−yA)/_A1. # i.e., Dy/Dx or Dy/_A1

[0056] For _A3: {square root}(Dx²+Dy²) or {square root}(_A1²+Dy²).

[0057] For _A4: yA−(Dy/Dx)xA, yA−_A2 xA, or yB−A2 xB. # 3 formulas

[0058] For _A5: ½(yB+yA). #=yM; similarly xM=½(xB+xA)

[0059] For _A6: (x−xM)²+(y−yM)²=¼(Dx²+Dy²), (x−xM)²+(y−_A5)² =¼_A3²,

[0060] (x−xM)²+(y−yM)² =¼_A3², or (x−xM)²+(y−_A5)²=¼(Dx²+Dy²).

[0061] Parameter Restrictions xA, yA, xB, yB are integers, with furtherrestrictions:

[0062] xA between 1 and 6; xB between −1 and −6 (xB not equal −A);

[0063] yA between 1 and 8; yB between 1 and 8 (yB not equal yA).

[0064] The first formula given for each of _A1, _A2, . . . , _A6 usesonly the problem's parameters; all others involve prior answers, denotedin italics as _A1, _A2, . . . , _A5.

[0065] Parameter restrictions are needed to ensure answers that are“typical” and non-trivial (e.g., not 0 or 1), and to avoid unacceptableoperations like division by zero when using the formulas. In Example 1,the restrictions on xA and xB ensure that any correct answer to _A1(i.e., Dx) is strictly negative, hence that division by zero cannotoccur in calculating _A2. Similarly, the yA and yB restrictions ensurethat Dy and the correct answers to _A2 and _A5 are not zero. Theparameter restrictions also avoid trivial addition or subtraction (ofzero) in getting _A3, _A4 and _A6.

[0066] The diagram in FIG. 2 illustrates Example 1 with xA=2, yA=1,xB=−1, and yB=3. With these values, which satisfy the parameterrestrictions, Dx=xB−xA is −3 (not 3) and Dy=yB−yA is 2. So the correctanswers for these parameters are:

[0067] _A1 is −3; _A2 is −⅔;

[0068] _A3 is {square root}13 (the square root of 13); _A4 is {fraction(7/3)};

[0069] _A5 is 2; _A6 is the equation (x−½)²+(y−2)²={fraction (13/4)}.

[0070] Typical intermediate calculations used to get these values are

[0071] Dx²+Dy²=(−3)²+(2)²=9+4=13 and yB+yA=3+1=4.

[0072] In FIG. 2, all formulas for a given answer produce the samecorrect value when correct answers are substituted for _A1, _A2, . . . ,_A5. However, the inventive system can grade problems for whichdifferent formulas for an answer give different correct values.

[0073] The Prior- and Later-Answer Strategies are implemented asfollows: Suppose one or more formulas for _Aj depend on a prior answer_Ai (i.e., where i is less than j).

[0074] Later-Answer Strategy: Do the following for each _Ai that appearsin a formula for _Aj: If an answer for _Ai was submitted and found to beeither correct or “acceptably” incorrect, then substitute it for _Ai inall _Aj formulas. Otherwise, try substituting all possible _Ai-formulavalues (include the intended correct ones that just depend on theparameters) for _Ai. Find all _Aj-formula values that result from allpossible combinations of all such substitutions for prior _Ais, andgrade the submitted _Aj answer as correct if it agrees with any valuethus obtained.

[0075] Prior-Answer Strategy: If a submitted _Aj answer is graded ascorrect, then do the following for each _Aj formula with which thesubmitted _Aj answer agrees: Grade all unsubmitted prior _Ais in theformula as correct, but leave those that were submitted along with _Ajas previously graded.

[0076] To illustrate, consider a student using the above parameters forExample 1. Suppose the student submits the incorrect answer −1 (ratherthan −3) for _A1, and −2 for _A2, and that −1 is an “acceptablyincorrect” answer to _A1. Since −2 is the value obtained by substituting_A1=−1 correctly in the second _A2 formula, the Later-Answer Strategydictates that _A2=−2 should be graded as correct; but _A1=−1 shouldremain as not-yet-correct. When the student next wishes to submitanswers, the Answer-Entry Window will look like FIG. 3, where theproblem statement appears in the usual mathematical notation (say usingthe MathML enhancement of HTML) and with parameters replaced by theactual values used. Thus, “A(xA, yA) to B(xB, yB)” in the problem textof FIG. 2 appears as “A(2, 1) to B(−1, 3)” in FIG. 3.

[0077] The Answer-Information Column of FIG. 3 indicates that thestudent previously entered answers for _A1 and _A2, but only _A2 wasgraded as correct. The shaded _A2 box indicates that it is disabled, toprohibit any attempt to re-submit an answer for _A2. In the CRO column,_A2 is shown as C (for correct), _A4 and _A6 are shown as R (forrequired) because no not-yet-correct answer has a formula in which _A4or _A6 appears, and _A1, _A3 and _A5 are shown as O (for optional)because a correct _A3 can be obtained using _A1, and a correct _A6 canbe obtained using _A3 and _A5.

[0078] Suppose that the student entered only the four answers shown inthe Answer-Entry Column of FIG. 3, namely the correct answer to _A4,incorrect (but “acceptable”) answers to _A1 and _A5, and an answer to_A6 that is incorrect, but was obtained by correctly substituting thevalue of the second _A3 formula for _A3 and the submitted _A5 answer for_A5 in the second formula for _A6. (It is assumed that some method isavailable for entering symbolic answers such as the equation for _A6, orat least verifying that an entered symbolic answer is syntacticallycorrect.) When the student clicks on the Grade Entered Answers button,the grading module of the inventive system first determines that thesubmitted _A1 answer is incorrect but “acceptable.” It then finds thatthe submitted _A4 answer agrees with the value obtained by all three _A4formulas when all possible correct _A2 values (here, only −⅔) aresubstituted for _A2 in the last two. Had _A2 not already been graded ascorrect, it would have been so graded by the Prior Answer Strategy.

[0079] The grading module then finds that the submitted _A5 answer failsto agree with any calculated _A5 value (here, only 2); so it leaves _A5as not-yet-correct. Finally, after determining that the submitted _A5answer, 4, is “acceptably incorrect,” it substitutes 4 for _A5 and allpossible _A3 values for _A3 (namely, the intended correct value {squareroot}13 from the first _A3 formula, and {square root}[(−2)²+(2)²]={square root}8 from the second) in all four _A6 formulas, and findsthat the submitted _A6 equation agrees with that of the second formula(with _A3={square root}8) but no others. So the Later-Answer Strategydictates that _A6 should be graded as correct. And since the _A6 formulavalue that agreed with the submitted _A6 answer used _A3, thePrior-Answer Strategy dictates that the unsubmitted _A3 should be gradedas correct. It then sees that the agreeing _A3 value depends on _A1, butdictates that the submitted _A1 be left as previously graded, in thiscase as incorrect.

[0080]FIG. 4 shows the updated Answer-Entry Window after the studentclicked on the Grade Entered Answers button to have the four submittedanswers in FIG. 3 graded as above, i.e., with maximum credit given forcorrectly-obtained intermediate answers, even if actually incorrect. Thenumber of tries were appropriately incremented, all but _A1 and _A5 aredisplayed as correct, and _A1 and _A5 are displayed as required becauseall answers with formulas that depend on _A1 and _A5 have been graded ascorrect.

[0081] After reviewing the returned results, the student will click oneither the (Re-)Enter Answers button (which will clear the previouslyentered answers and modify the Answer-Information Column to be as inFIG. 3) to re-submit answers for _A1 and/or _A5, or the Finished WithProblem button to return to a previous web page and get optionsregarding what to do next. Although not shown, hints and other guidanceand messages for each answer can be shown in the table'sAnswer-Information Column.

[0082] The student described above might subtract in the wrong order toget Dx=+3 and Dy=−2 (both with the wrong sign), but still got thecorrect _A2 answer, −⅔. If this student initially submits the correctanswer for _A2 but doesn't submit the incorrect Dx for _A1, then thePrior-Answer Strategy as described above would grade _A1 as correct. ThePrior-Answer Strategy can be adapted so that it is enabled or suppressedfor each answer rather than for the whole problem. Similar adaptationsof Later-Answer Strategy are possible, and subsumed in the teachings ofthis invention.

[0083] A second example involving symbolic parameters and answers willnow be used to illustrate the ease with which the inventive system cancreate problems with flexibility and educational capabilities that farexceed existing grading art. The problem text, correct-answer formulas,and parameter restrictions for Example 2 are set forth below. Example 2is a 4-answer calculus problem with 8 parameters, of which three (f, g,and h) are non-numeric, i.e., symbolic. Only two parameters actuallyappear explicitly in the problem text and formulas, namely f (thefunction to be differentiated), and xc (the x-coordinate of the tangencypoint). The remaining six, namely g, h, m, n, a, and b, are used tocreate f(x) as a sum, reciprocal, product, quotient, or composite ofthe; most-used elementary functions of a standard two-semester calculuscourse.

EXAMPLE 2 8 Parameter Calculus Problem

[0084] Problem Text

[0085] The derivative of y=f(x)=|* f(x) *| is f′(x)=|* ‘diff(f(x),x)’*|=_A1. At x=xc, f(xc)=_A2, f′(xc)=_A3, and the equation of the tangentline is y=_A4.

[0086] Enter _A2 and _A3 exactly, i.e., use sin 1, not 0.84 . . . ;|*sqrt(3) *|/2, not cos |* Pi *|/6 or 0.866 . . . ; etc.

[0087] Formulas for Answers [_Ai Denotes a Prior Answer Used to get _Aj]

[0088] For _A1: df(x)/dx. # i.e., dy/dx=f′(x) [derivative of f(x)]

[0089] For _A2: value of f(x) at x=xc. # i.e., f(xc)

[0090] For A3: value of df(x)/dx at x=xc or

[0091] value of _A1 at x=xc. # i.e., f′(xc)

[0092] For A4: f(xc)+f′(xc) (x−xc) or _A2+f′(xc) (x−xc) or

[0093] f(xc)+_A3 (x−xc) or _A2+A3 (x−xc).

[0094] Note 1: The formulas for _A1, _A2, and _A3 require single CAScommands. (See the CAS code for answers( ) below).

[0095] Parameter Restrictions xc is a non-decimal constant, and f(x) isa function of x subject to:

[0096] f(x) can be ag(x)+bh(x), 1/(ag(x)+bh(x)),

[0097] ag(x)h(x), ag(x)/h(x), or a g(h(x));

[0098] g(x) can be x^(n), sin(nx), cos(nx), e^(nx), ln(n+x);

[0099] h(x) can be x^(m), sin(mx), cos(mx), e^(mx), ln(m+x);

[0100] m and n are exact integers or fractions;

[0101] a and b are exact integers or fractions such that

[0102] f(xc) and f′(xc) are defined but not 0 or 1.

[0103] Note 2: With these values, the Problem Text will appear in theAnswer-Entry Window as:

[0104] The derivative of y=f(x)=x³+½x² is f′(x)=f(x)=_A1. At x=1,f(1)=_A2, f′(1)=_A3, and the equation of the tangent line is y=A4. Enter_A2 and _A3 exactly, i.e., use sin 1, not 0.84 . . . ; ½{square root}3,not cos ∂/6 or 0.

[0105] In Note 2, |* f(x) *| got displayed with parameter valuessubstituted; however, |* ‘diff(f(x),x)’ *| did not because the CAS Mapledoesn't substitute values in expressions enclosed in single quotes.Other CASs have comparable capabilities. The conversion fromauthor-provided problem text to HTML displayed in the Answer-EntryWindow is described in section 3.E.

[0106] The parameter restrictions of Example 2 look rather complicated,but can be handled easily by a general CAS test procedure that acceptssets of parameter values and restrictions as input and returns true orfalse depending upon whether or not the set of parameter values meet theset of restrictive conditions.

[0107] When creating a course activity that involves random selection ofparameters, e.g., a tutorial or ID-individualized homework, theInstructor Client allows the instructor to strengthen the parameterrestrictions imposed by the problem author. For Example 2, theinstructor might wish to allow only g(x)=x^(n) and h(x)=x^(m) or onlyg(x)=sin(nx) and h(x)=cos(mx), where and n can be only integers or onlyfractions. This enables the instructor of an introductory calculuscourse to (re-)use a single problem like Example 2 to grade just aboutany basic derivative a student might be asked to find on a test,tutorial, or homework assignment. Similarly, a single, versatile problemcan be (re-)used for definite or indefinite integration in calculus, ormore generally for any topic in any formula-based math, science, orengineering course.

[0108] In Example 2, the problem author provided four formulas for _A4,to allow for all four possible combinations of correct or incorrectanswers to _A2 and _A3. This was also done for _A6 in FIG. 2. In fact,the inventive system will check all such possible combinations as partof the Later-Answer Strategy, sparing the problem author the need to doso. Thus, only the first and fourth _A4 formulas were actually needed inExample 2, and only the first and second _A6 formulas were actuallyneeded in FIG. 2.

[0109] The parameter values used for the illustrative diagram of FIG. 5satisfy the parameter restrictions. With these values, the correctanswers to Example 2 are

[0110] _A1 is 3x²+2x, _A2 is {fraction (3/2)}, _A3 is 5, _A4 is{fraction (3/2)}+2(x−2),

[0111] with _A2 obtained as (1)³+(1)²/2, and _A3 obtained as 3(1)²+2(1).The student may submit the _A1 answer as 2x+3x² or in factored form asx(3x +2), x(2+3x), (2+3x)x, etc., or the _A4 answer in expanded form as−½+2x or 2x−½. When an answer is a symbolic expression such as _A1 or_A4, the problem's author must specify whether “agreeing with” anacceptable correct value means any permissible algebraic rearrangement,as one would want in for _A1 and _A4, or in a particular form such asfactored or expanded, as one might want for an introductory algebraproblem. The inventive system must be implemented with pre-definedcomparisons that enable various criteria for “agreement” of two decimalvalues (so many decimal places or significant digits), two equations(acceptable rearrangements, as for _A6 of Example 1), etc., as well asfor two algebraic expressions. Provision for such varied “agreement”comparisons is easily implemented using a computer algebra system (seethe discussion below of Comparison_Choices and of Selected_Comparisons).

[0112] This section describes how information flows when the effectivegrading described in the preceding sections takes place. Reference ismade to FIG. 1. For the Problem Table of the database, data for a givenproblem is provided by the problem's author via an Authoring Tool andthe Database Manager. For the Course Table, course activity datadescribing tests, tutorials, and homework assignments is provided by thecourse's instructor via the Instructor Client; and student performancedata recording the grading results is stored by the Student Client.

[0113] The author of a multi-answer problem must provide administrativedata (intellectual property ownership, contact information, etc.) alongwith the following grading data:

[0114] Parameter_Names A list of the names of the problem's parameters;

[0115] Problem_Text A CAS character string used to print the problemstatement or to view it via the Student or Instructor Client;

[0116] answers( ) A CAS procedure that describes all acceptable formulasand methods that can be used to get correct answers;

[0117] Parameter_Restrictions A set of conditions that limit the rangesof parameter values.

[0118] Acceptability_Tests A list of tests to ensure that an incorrectsubmitted answer yields “acceptable” values when used correctly to getlater answers (for the Later-Answer Strategy).

[0119] Comparison_Choices A list of acceptable ways to compare asubmitted answer with a “correct” value.

[0120] To illustrate this information, refer to Examples 1 and 2 (FIGS.2 and 5 and the problem specifications).

[0121] For Example 1, Parameter_Names stores the four-entry list [xA,xB, yA, yB]. For Example 2, it stores the eight-entry list [f, xc, g, h,m, n, a, b]. These lists become

[0122] [2, 1, −1, 3] and [x→x³+½ x², x→x^(n), x →x^(m), 3, 2, 1, ½],

[0123] respectively, when the parameter values used to get FIGS. 2 and 3are substituted.

[0124] Possible Maple procedures answers( ) for Examples 1 and 2 are:

[0125] answers:=proc(xA, yA, xB, yB) # for Example 1

[0126] local Dx,Dy, m, xM,yM, d2AB,d2PM,d2PA5; # d2=dist squared

[0127] Dx,Dy,m:=xB−xA, yB−yA, Dy/Dx;

[0128] xM,yM:=(xB+xB)/2, (yB+yA)/2; # midpt

[0129] d2AB,d2PM:=Dx{circumflex over ( )}2+Dy{circumflex over ( )}2,(x−xM){circumflex over ( )}2+(y−yM){circumflex over ( )}2; # P=(x,y)

[0130] formulas(1, [Dx]); # x_change

[0131] formulas(2, [m, Dy/_A1]); # slope

[0132] formulas(3, [sqrt(dAB2), sqrt(_A1{circumflex over( )}2+Dy{circumflex over ( )}2)]); # dist(AB)

[0133] formulas(4, [yA−m*xA, yA−_A1*xA, yB−_A1*xB]); # y-cept

[0134] formulas(5, [yM]); # y-mid

[0135] d2PA5:=(x−xM){circumflex over ( )}2+(y−_A5){circumflex over( )}2; # P on circle

[0136] formulas(6, [d2PM=d2AB/4, d2PA5=d2AB/4,

[0137] d2PM=_A3{circumflex over ( )}{fraction (2/4)},d2PA5=A3{circumflex over ( )}{fraction (2/4)}]); # circle'seq

[0138] end:

[0139] answers:=proc(f, xc) # for Example 2

[0140] local fc,Dfc;

[0141] fc,Dfc:=f(xc), D(f)(xc);

[0142] formulas(1, [D(f)(x)]); # f′(x)

[0143] formulas(2, [fc]); # f(c)

[0144] formulas(3, [Dfc, eval(_A1,x=xc)]); # f′(c)

[0145] formulas(4, [fc−Dfc*(x−xc), _A2−Dfc*(x−xc),

[0146] fc−_A3*(x−xc), _A2−_A3 *(x−xc)]); # y_tan

[0147] end:

[0148] The problem author provides correct-answers formulas for the jthanswer _Aj using a statement of the form

[0149] formulas(j, [Fj_(—)1, Fj_(—)2, . . . , Fj_Nj]),

[0150] where the first formula Fj_(—)1 depends only on the parametersand any additional formulas Fj_(—)2, Fj_(—)3, . . . generally depend onprior answers _Ai (see FIGS. 2 and 5). When procedure answers( ) isexecuted, it calls procedure formulas( ) with j=1, then 2, . . . , thenNj. If an answer for _Aj is submitted, then the procedure formulas( ) inthe grading module sees if it agrees with all possible values generatedby Fj_(—)1, Fj_(—)2, . . . , Fj_Nj and it then uses the Prior- andLater-Answer Strategies to update the correctness status of _A1, . . . ,_Aj (see ???). Procedure answers( ) is executed with a differentprocedure formulas( ) when a problem is installed the on the ProblemTable.

[0151] An instructor can set up a gradable activity so that thesubstituted parameter values are the same for all students (say, for aquiz or exam); individualized for each student, depending on a studentID stored on the database (say, for a homework); or assigned randomly(say, for a tutorial). For all three choices, limits must be imposed onboth assigned parameter values and incorrect submitted intermediateanswers to ensure reasonable answers and avoid run-time errors such asdivision by zero when correct or incorrect prior answers are substitutedin a formula for a later one.

[0152] The imposition of such restrictions is the responsibility of theproblem author, who does this by providing the setParameter_Restrictions and lists Acceptability_Tests andComparison_Choices, as described above. Parameter restrictions forExamples 1 and 2 are discussed hereinabove.

[0153] Acceptability tests are similar to parameter restrictions. ForExample 1, the problem author might specify the followingAcceptability_Tests list, in which [ ] indicates no restriction:

[_A1<0, [ ], _A3 >0, A4 >0, A5 is of mx+b form, [ ]].

[0154] _A3 is a nonzero length and therefore must be positive. The _A4test follows from the Example 1 restriction that yA and yB must begreater than 1, hence positive. The _A5 test demands that the answerhave the form of a straight line. This is easily done in a CAS, e.g., byrequiring that the Maple command degree(_A5,x) returns 1. Finally, _A2and _A6 require no test because no _Aj formula depends on _A2 or _A6.(In fact, the last answer of any problem cannot be a prior answer, hencenever requires an acceptability test.) For Example 2, the problem authormight require that _A1 has the same form as the derivative of f(x), say,a weighted sum of sin(3×) and cos(2×), and that _A2 and _A3 be neither 0nor 1. These conditions are also easily tested using a computer algebrasystem.

[0155] A reasonable Comparison_Choices list for Example 1 would specifyexact,- numerical equality for _A1 through _A5, and equivalent equationsfor A6. A reasonable list for Example 2 would specify equivalentexpression for _A1 and _A4, and exact numerical equality for _A2 and_A3. For both problems, the nature of the correct answers, allowed onlyone natural comparison for each one. Had some answers been decimals, theproblem author could have allowed the instructor to choose betweendecimal-place and significant-digit agreement between the submittedanswer and “correct” values generated by formulas( ).

[0156] To ensure reliable grading, problems must be added to thedatabase's Problem Table via an Authoring Tool that performs extensivetests on the author-provided data and, if ok, then sends it in thecorrect format to the Database Manager, who has the necessary utilitiesand privileges to add the problem to the Problem Table (see FIG. 1).

[0157] As part of the installation, either the Authoring Tool or theproblem-adding-utility will execute procedure answers( ) using aprocedure formulas( ) that augments the author-provided data withdependent-answer data that expedites the “bookkeeping” needed toimplement the Prior- and Later-Answer Strategies and the CROdeter-mination for the Answer-Entry Window. One way to do this is tocreate and store two lists of lists, say Dependent_Answers andAll_Prior_Answers, as follows:

[0158] When formula(j, [Fj_(—)1, Fj_(—)2, . . . , Fj_Nj ]) is executed,it stores the lists of all _Ais (if any) that appear in each of theformulas Fj_(—)1, Fj_(—)2, . . . , Fj_Nj as the jth entry ofDependent_Answers, and it stores the lists of all prior _Ais on whicheach formula Fj_(—)1, Fj_(—)2, . . . , Fj_Nj depends as the jth entry ofAll_Prior_Answers.

[0159] In Example 1, F2_(—)2 (yB−yA)/_A1and F3_(—)2={squareroot}(_A1²+Dy²) depend on _A1. Similarly, F4_(—)2 and F4_(—)3 depend on_A2; F6_(—)2 and F6_(—)3 depend on _A3; and F6_(—)2 and F6_(—)4 dependon _A5. Consequently, the 6-list list Dependent_Answers is

[0160] [[ ], [[ ],[_A1]], [[ ],[_A1]], [[ ],[_A2],[_A2]], [ ],

[0161] [[ ],[_A5,_A3],[_A3],[_A5]]]

[0162] and the 6-list list All_Prior_Answers is

[0163] [[ ], [[ ],[_A1]], [[ ],[_A1]], [[ ],[_A2, _A1]],[_A2, _A1]], [],

[0164] [[ ],[_A5,_A3,_A1]],[_A3,_A1]],[_A5,_A3,_A1]]]

[0165] Similarly, Example 2 has four answers, so Dependent_Answers is

[0166] [[ ], [ ], [[ ],[_A1]], [[ ],[_A2],[_A3],[_A3],[_A2]]]

[0167] and All_Prior_Answers is

[0168] [[ ], [ ], [[ ],[A1]], [[ ],[_A2],[_A3,_A2],[_A3,_A2]]].

[0169] The last entry of all these lists is a 4-entry list. For Example1 these four entries correspond to the four _A6 formulas [F6_(—)1,F6_(—)2, F6_(—)3, F6_(—)4]; for Example 2 they correspond to the four_A4 formulas [F4_(—)1, F4_(—)2, F4_(—)3, F4_(—)4]. Storing the _Aientries in reverse order improves the efficiency with which theyexpedite the Prior- and Later-Answer Strategies and C-R-O determination.

[0170] The Grading Module will use a different procedure formulas( ) todo the actual grading as prescribed by the problem author andinstructor. The use of a single procedure name within answers( ) [hereformulas( )] for different tasks, all transparent to the problem author,is seen as a non-obvious enhancement of existing computerized-gradingart.

[0171] It is assumed that a mechanism exists for storing an ID for eachstudent in each course on the Course Table. In assigning a problem for acourse activity, the Instructor Interface will ask the instructor tochoose a parameter selection method to indicate how values for aparameter-value list Parameter_Values are to be assigned. The choicesare: Instructor_Specified (Parameter_Values stores these values, used byall students); Random (the stored Parameter_Values will be randomlygenerated for each student); ID_Individualized (Parameter_Values willstore values determined by a student ID).

[0172] The parameter selection method is stored with the activityinformation on the course table. Provision must be made to ensure thatregardless of how the Parameter_Values list assigned, the values satisfythe author-provided parameter restrictions. For the parameter valuesgiven for Examples 1 and 2 as illustrated in FIGS. 2 and 5, these listsare

[0173] [2, 1, −1, 3] and [x→x³+½ x², x→x^(n), x→x^(m), 3, 2, 1, ½],

[0174] respectively, with symbolic entries stored using appropriate CASinstructions.

[0175] To simplify the remaining discussion, let us assume that whenseveral comparison choices for an Aj are available, the instructor willbe asked to select one, hence that each submitted answer is tested forcorrectness in only one way and the selected comparison descriptors arestored in a list Selected_Comparisons on the Course Table.

[0176] If the parameter selection method for an activity is Random orID_Individualized, then students will work problems with differentParameter_Values lists. If the list for a given student was notpreviously stored, the Student Client gets it from the CAS module andstores it on the Course Table. After grading, it also stores the listStatus, whose jth entry, denoted as Status_j, stores the student'sperformance data for answer _Aj as follows:

Status_j=0 (_Aj not tried), −n (n failed tries), or +n (correct on the nth try).

[0177] This important data will be used in the Answer-Entry Window andto generate reports for the student, instructor, and problem owner.

[0178] The Grading Module has three critical CAS components, hereinreferred to as prepare( ), getCRO( ), and grade( ). In describing them,it is assumed that the Student Client has performed the initial“handshaking” [viz., verify the student's ID for the course, then askfor a gradable activity (homework, tutorial, quiz/exam, etc.), a desiredproblem for the activity, and perhaps whether the student wishes to seethe problem statement displayed], and has queried the system database toget the data that is needed to grade the problem as described above.

[0179] The Student Client first passes the parameter selection method,student ID, lists Parameter_Names, Parameter_Values,Parameter_Restrictions, All_Prior_Answers, and Status, and Problem_Text(or a blank string if the problem statement is not wanted) to procedureprepare( ), which does the following:

[0180] Step 1. If the Parameter_Values list doesn't contain values, theyare assigned in accordance with the parameter selection method, subjectto the conditions in Parameter_Restrictions.

[0181] Step 2. Call procedure getCRO( ), which uses lists Status andAll_Prior_Answers to get the list CRO as described above.

[0182] Step 3. If passed Problem_Text rather than a blank string, callprocedure Convert2HTML( ) to get HTML that displays the problemstatement with Parameter_Values substituted for Parameter_Names.

[0183] Step 4. Return CRO, Parameter_Values, and HTML for the problemstatement to the Student Client.

[0184] Note 1: CASs have the built-in ability to convert CAS statementsto MathML (an HTML enhancement that displays mathematical expressions).This makes it easy to create a CAS procedure, say Convert2HTML( ), thatparses Problem_Text, substitutes Parameter Values for Parameter Names,and then converts the result to HTML that displays |* *|-delimited CASmath expressions in standard math notation.

[0185] Note 2: For some course activities, the student can choose tosubmit a file of answers created off-line using a provided softwaretool. In this case, the call to prepare( ) will not be necessary.

[0186] Recall that the jth entry of the list CRO is C, R, or O accordingas _Aj is already correct, required for full credit (i.e., a finalanswer), or optional (i.e., an intermediate answer). The Student Clientuses the lists CRO and Status to display the Answer-Entry Window (seeFIG. 3). It then receives the set of submitted answers and forms asubmitted-answer list SA, whose jth entry is either the submitted answerto _Aj or a not-answered designator, say na. For the submitted answersshown in FIG. 2, the submitted-answer list would be

SA=[−2, −⅔, na, na, {fraction (7/3)}, 4, (x−½)²+(y−4)²=2],

[0187] with CAS syntax used to describe the mathematical operations(here, squaring). The Student Client then passes SA, along with thelists Status, Parameter_Names, Parameter_Vaues, Dependent_Answers,All_Prior_Answers, Selected_Comparisons, and procedure answers( ) toprocedure grade( ), which grades the SA entries, updates Status toreflect the results, calls getCRO( ) with this revised Status list, andreturns the updated Status and CRO. Finally, the Student Client usesthese lists to report the results to the student (see FIG. 4).

[0188] As such, an invention has been disclosed in terms of preferredembodiments thereof, which fulfill each and every one of the objects ofthe invention as set forth hereinabove, and provide a new and usefulprocess for computerized grading of formula-based multi-step problemsvia a web-interface of great novelty and utility.

[0189] Of course, various changes, modifications and alterations in theteachings of the present invention may be contemplated by those skilledin the art without departing from the intended spirit and scope thereof.

[0190] As such, it is intended that the present invention only belimited by the terms of the appended claims.

1. A method of computerized grading of formula-based multi-stepproblems, including the steps of: a) providing a memory means forstoring said problems and data to enableland facilitate assessment ofcorrectness of said problems; b) storing at least one formula-basedmulti-step problem in said memory means, a complete solution to saidproblem requiring calculation of at least one intermediate answer and asubsequent calculation using said intermediate answer to arrive at saidcomplete solution; c) providing a computer program that can gradesubmitted answers to the steps of said problem; d) providing a userinterface with selective access to said memory means and said computerprogram; e) providing a global computer network with which said userinterface can be connected to said memory means and said gradingprogram.
 2. The method of claim 1, wherein a correct answer to one ormore parts of said problem can be obtained by any of several formulas ormethods, possibly requiring different previous-part answers, andpossibly resulting in different values that can be graded as correct. 3.The method of claim 2, further including the step of providing saidgrading program with capability to grade said problems in accordancewith a prior answer strategy in which determination is made whether anintermediate solution provided by a user is correct, and, if so, givescredit for all prior answers employed to arrive at said intermediatesolution.
 4. The method of claim 3, further including the step ofproviding said grading program with capability to grade said problems inaccordance with a later answer strategy in which responses to subsequentsteps are examined and are given credit, if incorrect, where obtainedcorrectly from incorrect but reasonable answers to one or more earliersteps.
 5. The method of claim 2, further including the step of providingsaid grading program with capability to grade said problems inaccordance with a later answer strategy in which responses to subsequentsteps are examined and are given credit, if incorrect, where obtainedcorrectly from incorrect but reasonable answers to one or more earliersteps.
 6. The method of claim 2, wherein said step of providing saiduser interface includes the step of displaying an answer entry windowincluding areas to display information relevant to solving a saidproblem.
 7. The method of claim 6, wherein said window displays whichparts of a problem have been graded as correct and which remaining partsmust be so graded for full credit, and then allows a user to submitanswers to any or all remaining parts, whether or not required for fullcredit.
 8. The method of claim 2, wherein said storing step for at leastone said problem includes the step of providing data to enablecorrectness assessment entirely in a computer-algebra-system readableform.
 9. The method of claim 2, further including the step of providingan authoring tool to test data provided to enable correctness assessmentfor a problem for correctness and robustness under actual or simulatedgrading conditions before said enabling data is stored.
 10. The methodof claim 2, further including the step of providing data to enableassessment of correctness of a said problem, said enabling data thenbeing used to create further stored data to facilitate said assessment.11. The method of claim 2, wherein a said problem includes a pluralityof intermediate steps.
 12. The method of claim 2, wherein a said problemincludes at least one parameter that can be assigned values that aremathematical expressions.
 13. A method of computerized grading offormula-based multi-step problems, including the steps of: a) providinga memory means for storing said problems and data to enable andfacilitate assessment of correctness of said problems; b) storing aplurality of formula-based multi-step problems in said memory means, acomplete solution to each of said problems requiring calculation of atleast one intermediate answer and a subsequent calculation using saidintermediate answer to arrive at said complete solution, memory meansbeing accessed via a global computer network, said accessing stepincluding the step of providing an authoring tool from which saidproblems are received by said memory means; c) providing a globalcomputer network to which said memory means may be selectivelyconnected; d) providing said memory means with data to facilitate (1)assessment of correctness of proposed answers to said problem and (2)preparation of content of an answer entry window that includes areas todisplay a user's progress in solving a problem.
 14. The method of claim13, further including the step of providing said grading program withcapability to grade said problems in accordance with a prior answerstrategy in which determination is made whether an intermediate solutionprovided by a user is correct, and, if so, gives credit for all prioranswers employed to arrive at said intermediate solution.
 15. The methodof claim 13, further including the step of providing said, memory meanswith a further capability to grade said problems in accordance with alater answer strategy in which responses to subsequent steps areexamined and are given credit, if incorrect, where obtained correctlyfrom incorrect but reasonable answers to one or more earlier steps. 16.The method of claim 13, wherein said window includes display of whethera user's intermediate solution to a problem is or is not correct. 17.The method of claim 13, wherein said storing step includes the step ofproviding said data to enable correctness assessment entirely in acomputer-algebra-system readable form.
 18. The method of claim 13,wherein at least one of said problems includes a plurality ofintermediate steps.
 19. The method of claim 13, wherein at least onesaid problem includes at least one parameter that can be assigned valuesthat are mathematical expressions.
 20. The method of claim 19, whereinsaid at least one problem includes symbolic and numerical parameters.21. The method of claim 13, wherein said window displays which parts ofa problem have been graded as correct and which remaining parts must beso graded for full credit, and then allows the user to submit answers toany or all remaining parts, whether required for full credit or not. 22.The method of claim 13, further including the step of providing at leastone stored problem and all data necessary to enable assessment ofcorrectness of said problem entirely in a computer-algebra-systemreadable form.
 23. The method of claim 13, further including the step ofproviding data to enable assessment of correctness of a said problem,said enabling data then being used to create further stored data tofacilitate said assessment.
 24. The method of claim 13, wherein a saidstored problem includes one or more parameters that can be assignedsymbolic values.